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ABSTRACT 


Hand gesture recognition is a topic that is still investigated by many 
scientists for numerous useful aspects. This research investigated hand 
gestures for sign language number zero to nine. The hand gesture recognition 
was based on finger direction patterns. The finger directions were detected 
by a Leap Motion Controller. Finger direction pattern modeling was based 
on two methods: threshold and artificial neural network. Threshold model 
1 contained 15 rules based on the range of finger directions on each axis. 
Threshold model 2 was developed from model 1 based on the behavior 
of finger movements when the subject performed hand gestures. The ANN 
model of the system was designed with four neurons at the output layer, 
15 neurons at the input layer, seven neurons at the first hidden layer 
and 5 neurons at the second hidden layer. The artificial neural network used 
the logsig as the activation function. The result shows that the first threshold 


Threshold model has the lowest accuracy because the rule is too complicated and rigid. 
The threshold model 2 can improve the threshold model, but it still needs 
development to reach better accuracy. The ANN model gave the best result 
among the developed model with 98% accuracy. LMC produces useful 
biometric data for hand gesture recognition. 
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1, INTRODUCTION 

Nowadays, the way human operates machine continues to evolve away from a physical button 
or relay to a virtual control method [1]. The interaction between human and machine becomes more flexible 
and mimic the communication between humans. Some methods were developed to control machines such 
as eye control [2] and muscle control [3]. Hand gesture is one of the methods developed by numerous 
researchers not only for sign language but also to build interaction between humans and machines and flex 
sensors [4-7]. A user interface that can be operated by mid-air hand gestures will become more popular 
shortly because of the expansion in the virtual technology and smart devices [8]. A hand gesture controlling 
application for computers such as on a website, VLC player and PowerPoint was developed by [9]. 

Some sensors are being developed such as flex sensors [10], gyroscopes [11] and cameras [12] 
to detect human activities. These sensors can be used together using a data glove for the hand gesture 
recognition system. A data glove is a special function glove integrated with hand gesture sensors [13]. 
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The sensors are attached to the glove. The sensors measure physical variable of movements when the hands 
move [14].Camera sensor faced challenge to remove noise from background of the hand. The average 
accuracy of the system with plain background was 94%, but it was only 64% when the background 
was not plain [9]. 

Measurement identifiers for hand gestures were divided into seven groups in [15]. They were finger 
position, finger direction, finger velocity, palm position, palm velocity, palm normal plane, and hand 
direction. These variables could be detected well by the Leap Motion Controller (LMC).This device can turn 
the hands and fingers into input devices to control a computer as substitutes of a mouse and keyboard [16]. 
The LMC could detect hand gesture and be used for rehabilitation purposes [17]. The hand gesture was also 
able to control a robot through the LMC [18]. It proves that LMC has a good performance to get 
biometric data. 

An anthropomorphic gripper was controlled by hand gestures based on an LMC. The performance 
of the LMC in the detection of hand gestures to control anthropomorphic hand could match the performance 
of a data glove [19]. The accuracy of the leap motion sensor and the data glove in reading the bending angle 
was also investigated by [14]. The average percentage error for the bending angle of LMC was larger than 
the data gloves. But the performance of LMC based on accuracy to control soft finger was better than that 
of the data glove. 

The performance of the LMC to detect 10 gestures were studied. Its performance was also compared 
to Kinect devices [20]. The characteristics of fingers used in leap motion were fingertip distance, fingertip 
angles and fingertip elevations. On the other hand, Kinect had curvature and correlation. Support Vector 
Machine (SVM) method was implemented to recognize the hand gesture. The accuracy of the LMC was 
80.86% and the Kinect was 89.71%. It shows that the accuracy of the LMC was better than the data glove but 
lower than the accuracy of the Kinect. But, LMC was cheaper, lighter and smaller than the Kinect device. 

Object tracking [21], feature extraction and recognition [22] method are the main issue in a hand 
gesture recognition system. A Hidden Markov Model (HMM) was trained with the Baum-Welch algorithm 
using Left-Right Banded [23]. The average recognition rate of this system was 90%. Superpixel-Graph 
Mover’s Distance (SP-EMD) method remained the problem of human habits to perform hand gestures. These 
habits formed variations of hand shapes. A canonical superpixel-graph was developed by [24]. In this 
research, hand gestures were recognized by a real time system. The finger positions were the input for the 
system. The characteristics of finger movements on the X, Y, and Z-axis of the leap motion were investigated 
to build the model of hand gesture. Two models were developed: threshold model and artificial neural 
network and their performances were studied. 


2. RESEARCH METHOD 

This is a real time hand gesture recognition system. The inputs are hand gestures and the sensor 
is an LMC. The user interface between human and machine are made in the Processing Software. Users can 
directly see the interpretation of the machine on their hand gestures. 


2.1. Hand gesture and finger direction 

There were ten hand gestures investigated in this research as shown in Figure 1. They were the signs 
for number ‘zero’ to ‘nine’. The gestures were performed by the right hand of the operators. The side of hand 
seen by readers was the same side of the hand facing the leap motion infrared camera. The position 
of the hand was about 25 cm in front of the leap motion camera, as illustrated in Figure 2. The Y-axis 
of the sensor was toward the hand. The Z-axis was the opposite of the gravity direction and the X-axis 


pointed to the left side of the operators. 
(QO) (1) (3) (4) 
(5) (6) 








YY 


Figure 1. Hand gesture of sign language for number ‘0’ to ‘9’ 
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Figure 2. Hand position to the leap motion sensor 


The hand gesture in this research was recognized based on finger directions as the feature. 
The finger direction (o) depended on its trigonometric function of angle (9) about the X, Y and Z-axis. 
Thus, a finger had three directions. The direction ranged from 1 to -1.If the angle was O° than the finger 
direction was -1. If the angle was 180° than the finger direction was 1.The relationship between the angle 
and the direction was investigated by experiments in this research. 


2.2. Threshold method 

The threshold model was developed based on the minimum and maximum values of finger 
directions. The finger directions of five subjects were recorded when they performed number zero to nine. 
They demonstrated each number 80 times. So, there were 400 data for each number from five subjects. 
For those data, minimum and maximum finger directions were calculated. These values were set 
for the threshold condition. A gesture was only recognized if the finger direction ranged from minimum 
and maximum value. So, if a finger worked in 3 different axes, there were 15 rules from five fingers 
for a sign number. These rules were investigated to be simplified. 


2.3. Artificial neural network method 

The outputs of the system represented number zero to nine with the inputs from the right-hand 
gestures. To represent the numbers, four-bit binaries were designed for the neural network outputs (0, to 04). 
The binary number for zero was OOOO and the binary number for nine was 1001. The complete information 
for this representation is shown in Table 1. 

The directions of the five fingers as the features of the right hand became the inputs of the artificial 
neural network. There were three directions for each finger: the direction for X, Y, and Z-axis. 
Thus, the system which recognized the right hand with five fingers needed fifteen inputs (4; to 1,5). Two 
hidden layers were implemented between the input and output. The first hidden layer had 7 neurons (j,; to j7) 
and the second hidden layer had 5 neurons (k; to ks). Two hidden layers and output layer had bias (bj, by 
and b,), as shown in Figure 3. The activation function of the neuron was a logsig function. The artificial 
neural network was trained using Lavenberg-Marquardt and the performance of training was evaluated based 
on mean square error. 


Table 1. The target of ANN 
0 1 pi D 4 5 6 a 8 9 
Output OOOO 0001 0010 OO11 0100 0101 0110 O1ll 1000 1001 





Figure 3. ANN architecture for hand gesture recognition 
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2.4. Research procedure 

Five volunteers participated in this research to determine and train the model based on their hand 
gestures. The first experiment was to study the characteristic of hand gestures based on finger directions. 
Then, they performed gestures eight times for each number to establish finger directions for number zero 
to nine. So, there were 4000 gestures from five volunteers for the ten numbers. The finger directions were 
recorded for minimum, maximum and average points. Based on those statistical data, two threshold models 
were designed and the neural network model was trained. Three models of hand gestures were designed. 
Their performances were studied based on accuracy. The threshold model and neural network model were 
tested using 1000 data. 


3. RESULTS AND DISCUSSION 
3.1. Characteristic of finger directions 

The characteristics of finger directions were investigated based on two experiments. The first 
experiment is shown in Figure 4. The first experiment had two types of finger movements. Index finger, 
middle finger, ring finger and little finger were of the first type, as shown in Figure 4(a). The fingers started 
from the straight position (0°) than bent to 30°, 60°, 90°, 120°, 150°, and 180°positions. Thumb finger 
movement was of the second type, as shown in Figure 4(b). The directions of the thumb finger were recorded 
for 0°, 30°, 60°, 90°, 120° and 150°. 





(b) 


Figure 4. The finger gesture in experiment | for, (a) Index finger, middle finger, ring finger, 
and little finger, (b) Thumb finger 


The result shows that the finger directions based on experiment | have patterns on X-axis, Y-axis 
and Z-axis, as shown in Figure 5. Based on the X-axis results, thumb finger and index finger have different 
patterns compared to the other three fingers. When the fingers bent from 0° to maximum position, 
the direction of thumb finger increased when it decreased on the other fingers. When the thumb finger moved 
to the right side, the other three moved to the left of the hand. It caused an increment of direction 
for the thumb finger. The state was different for index finger on the X-axis. Its direction almost had 
no difference on all bending situation. It concluded the situation that the index finger stayed almost 
at the same position on the X-axis. 


Bulletin of Electr Eng & Inf, Vol. 9, No. 2, April 2020 : 473 — 483 


Bulletin of Electr Eng & Inf ISSN: 2302-9285 O 477 


The directions of fingers on the Y-axis have a clear pattern. Index, middle, ring and little finger have 
the same pattern. Their directions started at 0.0, reached the maximum point which was almost 1.0 and then 
decreased to 0.0. The 0.0 direction describes that the fingers are perpendicular to the Y-axis 
and the 1.0 direction means that the fingers are about 180° to the Y-axis.Especially for thumb finger, 
the direction started from 0.25 and reached the maximum position on 0.70. 

The most explicit linear relationship between hand gesture and finger direction can be gained 
on Z-axis. The O° position has -1 direction and the 180°position has | direction. However, in the case 
of the thumb finger, it has a small range of direction on the Z-axis because of the limitation of its movement. 
It is better to choose X-axis to detect the thumb finger movement because it has the largest linear area than 


other axes. The advantage of an extensive range linear area is to provide more accuracy to detect 
the finger angle. 


©— Index 
<— Ring 
*~ Thumb 
, —#— Middle 
\— : —o— Little 


Direction (X) 
Direction (Y) 
Direction (Z) 





Angle 


Angle Angle 


Figure 5. The relationship between gestures and directions of fingers for experiment 1 


The second experiment investigated the directions of fingers when they moved only on XZ-plane. 
The subjects pulled the finger together for the normal position. Then the finger spread apart in three steps: 
small, medium and big spreads. The hand gesture for this experiment is shown in Figure 6. 
Based on the result shown by Figure 7, the direction of fingers changed for thumb,ring and little finger with 
a linear trend.But, the index finger and middle finger almost stayed in the same direction on X-axis. 
The directions of all fingers on the Y-axis tended to be similar among all hand gestures. There was no impact 
of the gesture to the finger directions. It changed randomly but in a small range direction. It correlates to 
the direction of the finger which is always the same as the Y-axis which is 90°. The direction of this angle 
is 0.0. The same situation happened on Z-axis. The directions of fingers were almost the same for all 
gestures. The normal position shows that all fingers were about 0° from the Z-axis. Then four hand gestures 


only gave a small change on it. The relationship between hand gesture and finger direction with the linear 
method on this axis is indistinguishable. 





(a) (b) (Cc) (d) 


Figure 6. The finger gesture in experiment 2 for, (a) Normal, (b) Small spread, 
(c) Middle spread, (d) Big spread 
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Figure 7. The relationship between the gestures and directions of fingers for experiment 2 


3.2. Accuracy 
3.2.1. Threshold model 1 
This is the basic model. Five subjects performed hand gestures 10 times for number zero to nine. 
Three variables were calculated based on those data: minimum, maximum and average of finger directions, 
as shown in Figure 8. The minimum and maximum of the directions were used for the threshold model. 
In Method I, all fingers considered all axes. The resulting accuracy of this model is still low. Only numbers 
zero and four were detected with 60% accuracy by this model. But on average, the accuracy is only 28%. 
The worst condition happened with numbers | and 2 where this model failed to detect any gesture for this 
number. The complete data are shown in Table 2. 
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Table 2. The confusion matrix of threshold model 1 
; 0 1 2 3 4 5 6 q 8 9 Unknown Accuracy 
0 60 : : : : : : : : : 40 60% 
1 : 0 : : - : : : : : 100 0% 
2 : - 0 i - : é : : : 100 0% 
3 : : : 40 : : : 2 : : 60 40% 
4 : : : : 60 : : g : : 40 60% 
5 : : : : : 10 : : : : 90 10% 
6 : : : : : : 20 : : : 80 20% 
7 3 : : : : : : 30 - 3 70 30% 
8 : : : : : : : : 20 : 80 20% 
9 : 2 : : 2 : : : z 40 60 40% 


i=Input o=Output 


3.2.2. Threshold model 2 

This model was developed based on threshold model 1 by considering the patterns of directions 
based on the experiment | and 2. From those experiments, some fingers only had slight movement in specific 
axes. Thus, the rule was deleted for that situation. Table 3 was used to build threshold model 2. Only X-axis 
was considered to build the rule for the thumb finger. The little and ring finger had complete rules 
on the X, Y, and Z-axis. Nevertheless, the index and the middle finger only worked on Y and Z-axis. 
The accuracy of threshold model 2 is slightly better than the threshold model 1, as shown in Table 4. 
However, this method is not recommended for the hand gesture recognition system. This method had 
difficulty in anticipating the variety of human hand shapesdue to differences in habits of each person during 
hand movements [24]. 
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Table 3. Active axes to build threshold model 2 
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Table 4. The confusion matrix of threshold model 2 
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3.2.3. Artificial neural network 
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An artificial neural network was designed to recognize the hand gesture. The number of input was 
15. They were the directions of five fingers in the XYZ-coordinate system. There were two hidden layers. 
The first hidden layer had seven neurons and the second hidden layer had five neurons. There were 400 data 
for training and 100 data for testing. The result shows that the best validation performance was 2.48x10° 
at epoch 38. The correlation between the output and the target for training is 0.99. There is three weighted 
phases for this neural network: between input and the first hidden layer,as shown in Table 5, between 
the first hidden layer and the second hidden layer,as shown in Table 6 and finally between the second hidden 


layer and output layer,as shown in Table 7. 


Table 5. The weight from the input layer (1) to the first hidden layer (j) 
i 
2 


ly lb 1 lu 15 16 


ji Zi -2.5 0.7 1.4 0.1 -1.5 
jo -5.9 -1.3 Zl 0.8 0.1 -1.2 
j3 -1.4 0.3 -1.7 -0.7 -1.7 -2.2 


ja -1.8 0.5 -1.8 Se 2n1 2.2 
Js 1.6 2.8 0.9 -1.9 -2.0 0.7 
Jo 25 -0.5 1.8 -2.0 -0.7 0.3 
j7 0.8 0.2 1.3 -0.7 -1.2 -0.4 


-0. 
3, 
-0. 
4, 


3 
4 
2 
1 


3.9 
2.8 


lio 
0.6 
-4.9 
2.6 
2.0 
0.0 
-0.2 
-1.2 


lu 7) 113 
0.4 -0.6 -1.0 
-2.9 2.0 2.8 
0.5 -0.9 Be 
-0.4 0.0 -1.5 
2.4 2.0 -2.5 
0.4 -2.5 0.2 
0.1 1.7 3.6 


Table 6. The weight from the first hidden layer (j) to the second hidden layer (k) 


ji iD 
ky 3.1 1,2 
1) -5.5 -4.3 
k3 -2.4 -7.9 
ky 2.4 -4.0 
ks 1.9 0.8 


js 
-1.2 
-1.0 
8.6 
-0.6 
-4.8 


J4 
2A 
-1.1 


Js 
-3.5 
-3.0 
2 
5.6 
3.6 


Je 
-3.1 
6.5 
-2.5 
-5.7 
-4.2 


i 
-3.0 
-2.8 
-3.6 
-1.1 
-1.9 


Table 7. The weight from the second hidden layer (k) to the output layer (0) 


ki 
O] -1.6 
O2 -1.6 
03 2 
O4 0.5 


ky 
1.0 
-1.] 
-2.2 
-11.2 


k; 
-2.7 
6.1 
135 
-4.3 


ky 
-7.2 
5.0 
-5.0 
0.6 


ks 
-3.0 
-11.5 

1.8 

1.5 


The performance of ANN with 15 inputs and four binary outputs can solve the problem of accuracy 
in the threshold model. The ANN accuracy is 98% with 1000 hand gesture data for testing, as shown 
in Table 8. The only mistake occurred for gesture zero. The position of fingers of this gesture made the LMC 
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fail to calculate the hand gesture. As shown in Figure 1, the position of the palm for zero is different 
from the other gestures. The palm is perpendicular to the Y-axis for zero gesture but it is 180° for the other 
nine gestures. The thumb finger located at the farthest location to the origin of LMC. Other fingers hide it. 
On the other hand, the accuracy of LMC to detect the angle of thumb is the lowest than the accuracy to detect 
other fingers [14]. Accuracy is the challenge of hand gesture recognition using LMC. 

Table 9 shows the performance of some hand gesture methods. The hand gesture recognition system 
using LMC was developed by [17]. This research applied the Support Vector Machine (SVM) and K-Nearest 
Neighbor (KNN) to recognize the hand gesture. The accuracy of the system was 97.3% and 97.9%. The hand 
gesture recognition for the Arabic number was studied by [25]. The hand features were finger position 
and the angle on X and Y-axis. It has 2-fold and 10-fold methods to recognize the hand gesture. 
The performance was 94.2% and 95.1%. In this research, AAN can recognize 98.0% hand gesture 
for numbers. The difference between [25] and this research is the hand gesture used to represent the numbers. 
EOG [2] and EMG [3] and flex sensor [4]. 


Table 8. The confusion matrix of ANN model 
0 1 2 3 4 5 6 2 8 9 Unknown = Accuracy 


80 20 - - - - - - - - - 80% 
100 - - - - - - - - 100% 

100 - - - - - - - - 100% 

- 100 - - - - - - - 100% 

- 100 - - - - - - 100% 

- 100 - - - - - 100% 

- 100 - - - - 100% 

- 100 - - - 100% 

- 100 - - 100% 

- 100 - 100% 


OmAANINDNFPWNK OC 


i=Input o=Output 


Table 9. Accuracy of hand gesture recognition methods 


Object Sensors Method Accuracy 
ae SVM 97.3 % 
Seven gesture for post stroke rehabilitation [14] LMC KNN 07.9 % 
Hand gesture for Arabic number 0 to 9 [22] LMC RBF 10-fold 95.1% 
Hand gesture 0 to 9 (Proposed) LMC ANN 98.0 % 


4. CONCLUSION 

In this study, the gestures of the right hand are detected using LMC. LMC calculated the position 
of five fingers for sign language of number 0 to 9. Two methods for the recognition system were based 
on threshold and ANN. The threshold method was design based on the minimum and maximum finger 
directions. Recognizing hand gestures using the threshold model is very difficult. This method has difficulty 
accommodating the variety of hand shapes. The system needs more training data to improve the accuracy 
of this method. An ANN model can give better results with the number of data for training. An ANN with 15 
inputs and four binary outputs can raise the accuracy of the system up to 98%. The problem with LMC 
is the capability to detect the thumb finger for number zero because the other fingers hide it. 
The implementation of ANN to the features of fingers driven by LMC is an alternative method for human 
and machine interaction. 
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